About me 👋

Hypothesis testing and errors

When testing a null hypothesis H_0 against an alternative H_1, we can make two types of errors.

We (usually) control the Type I error rate at a pre-specified level \alpha (typically 0.05).

Frequentist hypothesis testing

You compare the scores of the control and experimental groups and obtain a p-value of 0.05.

Bayesian hypothesis testing

After comparing the scores of the control and experimental groups, you find that the posterior probability that the groups differ is 95%.

Bayesian methods in psychology

(Heck et al., 2023; Jevremov & Pajić, 2024; Van De Schoot et al., 2017)

Multiplicity

Multiplicity (or multiple testing problem) emerges when we test more than one hypothesis.

• I test the effect of the manipulation on both working memory and inhibitory control.

• I test the effect of the manipulation and its interaction with age on working memory.

• I run post hoc tests to examine differences across four time-points.

Why multiplicity creates problems

When testing m hypotheses independently, each at Type I error rate \alpha, the overall probability of at least one false positive (family wise error rate) grows rapidly with m.

Multiplicity in Bayes

• Bayes Factor (prior odds)

• Posterior Probabilities (joint credibility)

Bayes Factor

Bayes Factor

From Keysers et al. (2020)

Bayes Factor and multiple testing

\text{Pr(at least one } BF \geq 3)

N.B. Results change depending on priors and threshold.

Posterior odds and multiple testing

Researchers often interpret BF as posterior odds (Hoijtink et al., 2016; Tendeiro et al., 2024; Tendeiro & Kiers, 2019).

\text{Posterior odds, } \frac{Pr(\mathcal{H_1} | data)}{Pr(\mathcal{H_0} | data)} = \frac{Pr(\mathcal{H_1})}{Pr(\mathcal{H_0})} \times \frac{p(data | \mathcal{H_1})}{p(data | \mathcal{H_0})}

Multiplicity adjustment can be incorporated by modifying the prior odds \frac{Pr(\mathcal{H}_1)}{Pr(\mathcal{H}_0)} based on the number of hypotheses tested, which produces posterior odds that control for multiple testing.

Bayesian estimation

The prior (grey) represents our initial guess about plausible values; after observing data, it updates to the posterior (blue), which narrows around the most likely value.

Bayesian estimation concrete example

Suppose we want to estimate the Stroop effect (0.060–0.120s). We can adopt priors centered at zero with varying standard deviations (τ = 0.1, 0.25, 0.5) to reflect increasing skepticism.

Probability of Direction

p_d = \max(Pr({\hat{\beta}} < 0), Pr({\hat{\beta}} > 0))

Probability of Direction and Multiplicity

\text{Pr(at least one } P_d \geq 0.975)

The joint posterior approach

You want to test whether an intervention improves verbal and/or visuospatial abilities, identify which domains improve. Improvement in at least one domain is more probable than improvement in both.

Example of hypothesis testing

Questions for You

Who here has used Bayesian methods? What was your experience?

What are the main barriers to correct statistical understanding and practice in your field? What would help us improve?

We often test multiple hypotheses in a single study. What drives this practice in your research? Do the benefits outweigh the costs?

H_0 = 0, H_1 = Cauchy(scale = \sqrt{2}/2)

Between-models priors (Jeffreys, 1938)

• m alternative hypotheses.

• Let \mathcal{H}_A be the event that at least one alternative hypothesis is true.

• We assign equal prior probability to the event \mathcal{H}_A and to its complement \mathcal{H}_0 (the global null, all false):

Pr(\mathcal{H}_0) = Pr(\mathcal{H}_A) = \frac{1}{2}

This ensures that the probability of all nulls being true simultaneously and the probability of at least one alternative being true both equal 0.5.

Westfall’s approach for dependent tests

For multiple comparisons among m means, Westfall et al. (1997) developed a solution for the \binom{m}{2} possible pairwise comparisons involving \mu_1, \mu_2, \ldots, \mu_m, where \theta_{ij} represents the difference \mu_i - \mu_j. Their hierarchical prior structure specifies:

\mu_i \begin{cases} \equiv \mu & \text{with probability } \lambda, \\ \sim G & \text{with probability } 1-\lambda, \end{cases}

with G(\cdot) being a continuous distribution.

Under this specification, the probability that any two specific means equal \mu is \lambda^2, and \text{Pr}(\text{all }\theta_{ij} = 0) = \lambda^m. This dependency structure yields a modified adjustment k = 1-0.5^{2/m}, replacing the independence-based correction (resembles previously described mixture models).

Note: Implemented in JASP for post-hoc testing (for details, see de Jong, 2019).

Bayesian estimation prior precision